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EXECUTIVE  SUMMARY 


The  purpose  of  this  study  project  is  to  highlight  some  of  the  problem 
areas  encountered  by  Project  Managers  when  formulating  the  Coordinated  Test 
Plan  (CTP)  for  his  project.  The  Army's  "integrated  testing"  approach  has 
placed  special  emphasis  on  contractor  testing  over  the  last  few  years  in 
lieu  of  independent  Government  testing  and,  consequently,  the  important  role 
of  contractor  testing  has  moved  to  the  forefront. 

Experience  has  shown  that  bilateral  communications  problems  often  mate- 
rialize during  the  RFP/proposal  evaluation/contract  initiation  process  and 
present  unwarranted  challenges  for  the  Project  Manager.  The  contractor  must 
transform  RFP  test  requirements  into  a proposed  test  program  for  Source 
Selection  Evaluation  Board  (SSEB)  evaluation  and  integration  with  Govern- 
ment test  objectives  in  the  CTP.  Any  weaknesses  or  deficiencies  in  the 
contractor  test  plan  must  be  resolved  with  minimum  effects  on  program  cost 
or  schedule  and  herein  lies  the  problem. 

Constructive  recommendations  for  improving  the  test  coordination  process 
are  presented  with  emphasis  on  RFPs,  contract  proposals,  proposal  evalu- 
ation, personnel  continuity  and  managing  the  test  program. 
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Figure  1.  CTP  Outline  for  Missile  System  Development  . 


SECTION  I 


INTRODUCTION 


Purpose  of  the  Study  Project 

The  author  has  been  associated  with  testing  of  Army  materiel  for  approxi- 
mately eighteen  years,  seventeen  years  in  a Government  test  laboratory  and 
one  year  in  a Project  Management  Office.  This  experience  in  the  Materiel 
Acquisition  Process  has  shown  that  development  of  the  Master  Test  Plan  for 
the  Engineering  Development  acquisition  phase  often  poses  numerous  challenges. 
Normally  it  becomes  an  iterative  process  within  narrow  bounds  because  of  un- 
favorable effects  on  project  cost  and  schedule. 

The  primary  purpose  of  this  study  project  is  to  highlight  some  of  the 
problems  that  the  Materiel  Oeveloper  (Project  Manager)  has  typically  en- 
countered in  the  past  when  attempting  to  Integrate  contractor  testing  with 
Government  test  objectives  during  preparation  of  the  Test  and  Evaluation 
Master  Plan  for  Full  Scale  Engineering  Development,  The  secondary  purpose 
is  to  present  a series  of  constructive  recommendations  for  future  develop- 
ment efforts  in  the  hope  that  “lessons  learned"  from  the  past  may  be  bene- 
ficial In  the  future. 

Scope  of  the  Study  Project 

This  paper  presents  a discussion  of  procedures  previously  utilized  by 
Army  projects  for  developing  their  Master  Test  Plan,  some  of  the  problems 
encountered  and  apparent  underlying  reasons  for  these  problems.  In 
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conclusion,  constructive  suggestions  are  submitted  for  consideration  by 
future  program  managers.  Suggestions  presented  herein  represent  collective 
opinions  of  experienced  personnel  in  the  field  and  should  improve  the  Master 
Test  Plan  and  the  materiel  acquisition  process  if  properly  applied. 

Discussion  is  presented  on  the  Department  of  Defense  and  Department 
of  Arn\y  directives  and  regulations  which  provide  guidance  for  the  preparation 
and  contents  of  the  Program  Master  Test  Plan  as  a part  of  the  overall  "Single 
Integrated  Development  Test  Cycle  Policy  in  Development  Test  and  Evaluation 
of  Army  Materiel."8  Second,  interest  is  focused  on  that  portion  of  the 
Master  Test  Plan  which  covers  contractor  testing  of  components,  subsystems 
and  systems,  how  the  Government  utilizes  contractor  test  results,  and  some 
of  the  challenges  for  obtaining  definitive  test  plans  in  development  con- 
tracts. 

Specific  Goals  of  the  Study  Project 

The  specific  goals  of  this  study  project  are: 

A.  To  show  the  significant  role  and  importance  of  contractor  testing 
during  Full  Scale  Engineering  Development. 

B.  To  show  how  contractor  testing  interfaces  with  Government  tests  in 
the  Master  Test  Plan. 


8this  notation  will  be  used  throughout  the  report  for  sources  of 
quotations  and  major  references.  The  first  number  Is  the  source  listed 
in  the  bibliography.  The  second  number  is  the  page  in  the  reference 
when  applicable. 
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C.  To  point  out  some  of  the  problems  typically  encountered  by  Project 
Managers  when  definltlzed  contractor  testing  was  not  part  of  the  basic 
contract. 

D.  To  determine  if  guidance  provided  to  contractors  for  preparation 
of  contract  proposals  has  generally  been  sufficient  in  the  past  or  should 
the  Government  Request  for  Proposals  (RFPs)  be  more  specific. 

E.  To  determine  if,  in  general,  descriptions  of  the  test  program  in 
contract  proposals  provide  sufficient  detail  for  proposal  evaluation  and 
understanding  of  each  level  of  testing  (system,  subsystem,  component, 
environmental  qualification,  etc.}. 

F.  To  determine  if  the  contractor's  test  program  in  the  contract  pro- 
posal should  be  in  AR  70-21®  format  and  of  sufficient  detail  for  incorporat- 
ing directly  into  the  program  manager's  Master  Test  Plan,  hereafter  referred 
to  as  The  Coordinated  Test  Program  (CTP)  as  per  AR  70-21®. 

G.  To  determine  if  the  present  policy  of  requiring  contractors  to 
submit  their  Master  Test  Plan,  hereafter  referred  to  as  the  Engineer  Design 
Test  Plan*,  60  to  90  days  after  contract  is  the  optimum  approach. 

H.  To  develop  conclusions  and  constructive  recommendations  for  im- 
proving the  process  on  future  development  programs. 

Methods  of  Research 

Multiple  sources  of  information  were  utilized  for  this  study;  Govern- 
ment directive..,  regulations  and  publications;  periodicals  and  written 
literature;  Interviews  with  Government  and  contractor  personnel  who  are  or 
have  previously  been  involved  in  the  development  of  Master  Test  Plans; 
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and  personal  experience.  Persons  Interviewed  a**e  not  Identified  in  this 
report  since  the  nature  of  this  study  suggests  no  "authoritative  sources" 
and  information  was  obtained  on  a non-attribution  basis. 

A literature  search  was  conducted  within  the  available  time  constraint 
to  obtain  documents,  reports,  regulations,  periodicals  and  other  written 
materials  related  to  the  subject.  No  literature  was  found  which  addressed 
the  specific  subject  of  Interest;  however,  a great  deal  of  literature  has 
been  written  about  problem  areas  closely  related  to  the  areas  of  concern 
and  some  are  referenced  herein. 

Informal  interviews  were  conducted  with  experienced  test  personnel  in 
several  Government  and  contractor  Project  Management  Offices  and  functional 
laboratory  personnel  who  have  been  instrumental  in  the  development  of 
contractor  Engineer  Design  Test  Plans  and  Government  Coordinated  Test  Plans 
for  several  programs.  Representatives  from  twelve  Arry  programs  in  various 
stages  of  development  were  Interviewed  and  the  results  of  those  Interviews 
are  summarized  herein. 
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SECTION  II 


BACKGROUND 


Requirements  for  the  Master  Plan 


Department  of  Defense  Directive  5000.3,  Test  and  Evaluation6,  provides 
the  basic  guidance  for  the  Master  Test  Plan. 

"The  DOD  Component  will  prepare  as  early  as  possible  in  the 
acquisition  process,  and  prior  to  Initiation  of  Full-Scale  Develop- 
ment, an  overall  test  and  evaluation  plan  to  identify  and  inte- 
grate the  effort  and  schedules  of  all  T&E  to  be  accomplished  and 
to  insure  that  all  necessary  T&E  is  accomplished  prior  to  the  key 
decision  pointss".6*6 

The  Department  of  the  Amy  promulgated  the  DOD  policy  through  AR  1000-110 
and  AR  70- IQ6  and  issued  detailed  instructions  for  implementation  of  the 
TSE  Master  Test  Plan  Policy  through  AR  70-21®. 

Testing  is  conducted  to  demonstrate  how  well  the  materiel  system 
meets  its  technical  and  operational  requirements;  provide  data  to  assess 
developmental  and  operational  risks  for  decision  making;  verify  that  the 
technical,  operational  and  support  problems  identified  in  previous  testing 
have  been  corrected;  and  to  insure  all  critical  issues  to  be  resolved  by 
testing  have  been  adequately  considered.  All  testing  Is  of  interest  start- 
ing with  contractor  bench  testing  and  going  through  controlled  tactical 
exercises  by  the  User.10*6 

During  the  full-scale  development  phase  and  prior  to  the  first  major 
production  decision,  the  Development  Test  accomplished  shall  be  adequate  to 
insure  that  engineering  is  reasonably  complete;  that  all  significant  design 
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problems  (Including  compatibility,  inter-operability,  reliability,  avail- 
ability, maintainability  (RAM),  and  supportability  considerations)  have 
been  identified;  and  that  solutions  to  the  above  problems  are  in  hand.10,5 

Thus  the  importance  of  a thorough,  well  managed  and  coordinated  Develop- 
ment Test  Program  is  highlighted  by  the  Department  of  Defense  and  the 
Department  of  the  Army  in  their  basic  policy  guidelines. 

Previous  ISP  studies  have  been  conducted  to  investigate  the  success  and 
problems  associated  with  implementing  the  aforementioned  test  policy  regu- 
lations and  it  was  generally  concluded  that: 

A.  Interface  problems  were  encountered  between  the  Materiel  Developer 
and  the  Operational  Tester  in  the  early  implementation  phases  of  AR  70-21.9; 
however,  these  problems  are  being  resolved  through  education,  personnel 
turnover  and  experience.14 

B.  Implementation  of  the  Master  Test  Plan  (Single  Integrated  Develop- 
ment Test  and  Evaluation)  Concept  has  been  successful  within  the  Army  and 
highly  beneficial  to  new  programs  which  have  utilized  it  from  Inception 

or  from  early  stages  of  development. ^ 

Requirements  for  the  T&E  Master  Plan  have  therefore  been  clearly 
established  and  Implemented  within  the  Army, 

Relationship  of  Contractor/Government  Testing 

As  previously  noted,  DOD  Directive  5000. 36  requires  preparation  of  a 
Coordinated  Test  Plan  for  Engineer  Development  prior  to  DSARC  II  and, 
consequently,  prior  to  selection  of  a prime  contractor  for  system  development. 
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The  Coordinated  Test  Plan  is  normally  prepared  by  the  system's  Project 
Management  Office  during  the  Validation  and  Demonstration  (Advanced  Develop- 
ment) phase  with  Inputs  from  the  Operational  Tester,  the  Amy  Test  Evaluator, 
and  other  agencies  Involved  in  the  test  program.  Since  a prime  contractor 
for  development  has  not  been  selected  at  that  time,  the  Project  Manager's 
Office  provides  Input  to  the  Coordinated  Test  Plan  to  show  the  categories 
and  time  phasing  of  contractor  testing  he  will  require  during  Engineering 
Development.  For  purposes  of  illustration  and  discussion,  a typical  Coordi- 
nated Test  Plan  (CTP)  outline  for  the  development  of  a missile  system  may 
include  general  categories  of  tests  as  indicated  in  Figure  1.  Tests  indicated 
in  Figure  1 will  be  utilized  to  show  the  relationship  between  contractor 
and  Government  testing, 

Since  the  original  CTP  is  prepared  without  contractor  Involvement,  it 
is  necessary  to  update  the  CTP  to  Include  contractor  testing  after  the 
contractor  is  selected.  More  will  be  said  on  this  subject  later. 


I.  Development  Tests  (DT-II) 

A.  Component  Tests 


1.  Breadboard  Tests 

2.  Prototype  Development  Tests 

3.  Environmental  Qualification  Tests 

4.  Reliability  Tests 

B.  Safety  Tests 

C.  Electromagnetic  Radiation  Tests 

D.  Countermeasures  Tests 

E.  Human  Factors  Tests 

F.  Missile  System  Flight  Tests 

1.  Development  Prototype  Tests 

2.  Tactical  Prototype  Tests 

3.  Environmental  Qualification  Tests 

G.  Environmental  Storage  (Long-term)  Tests 
II.  Operational  Tests  (OT-II) 

Figure  1.  CTP  Outline  for  Missile  System  Development 
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Considerable  literature  has  been  published  over  the  years  concerning 
how  to  plan  and  manage  test  programs  for  materiel  acquisition;  however,  the 
literature  generally  addresses  testing  at  the  total  system  level  with  very 
little  discussion  about  testing  at  the  component  and  subsystem  levels.  These 
tests  are  very  much  a part  of  the  contractor's  test  program  and  represent 
major  thrusts  in  the  development  testing.  Component  and  subsystems  tests 
are  major  contributors  to  the  future  successes  and/or  failures  in  the  program. 

Until  recent  years,  the  Army  test  policy  for  Engineering  Development 
strongly  supported  Independent  Government  development  testing  as  well  as 
Independent  operational  testing.  Under  the  old  policy,  little  emphasis 
was  placed  on  the  details  of  contractor  testing  since  the  Government  obtained 
the  data  necessary  for  system  evaluation  from  the  Independent  Government 
tests.  More  recently,  however,  the  Army  test  policy  has  changed  toward 
"Single  Integrated  Development  Testing".11  This  concept  is  Intended  to 
reduce  developmental  test  time,  test  hardware,  test  costs  and  eliminate 
redundant  testing  by  Integrating  contractor  and  Government  testing  Into  a 
composite  test  program.  The  materiel  developer  can  thus  verify  that  a 
new  weapon  system  meets  all  of  Its  performance  requirements  with  a minimum 
of  testing. 

The  Single  Integrated  Development  Test  policy  essentially  eliminates 
independent  Government  testing  and  stresses  Independent  Government  evalu- 
ation. This  approach  permits  an  Independent  Government  evaluator  to  monitor 
both  contractor  and  Government  test  programs  and  evaluate  the  system  from 
the  results  of  the  combined  test  programs;  however,  the  contractor  tests 
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must  satisfy  Government  test  objectives  as  well  as  contractor  test 
objectives. 

With  the  Increased  emphasis  on  contractor  testing,  the  contractor's 
test  plan  becomes  a major  consideration  in  the  materiel  developer's  Coordi- 
nated Test  Plan. 

Under  the  Single  Integrated  Test  Policy,  all  of  the  Development  Tests 
(DT-II)  except  the  Environmental  Storage  Tests  (see  Figure  1)  would  typi- 
cally be  conducted  by  the  contractor  and  monitored  by  the  Government.  The 
contractor  Is  consequently  conducting  the  major  portion  of  the  Engineering 
Development  Tests. 

It  Is  not  uncommon  for  contractor  testing  to  be  conducted  at  a Govern- 
ment test  facility  since  contractors  often  do  not  have  adequate  facilities 
for  testing  hazardous  components  or  for  missile  flight  tests,  etc.  This 
facilitates  test  monitoring  by  Independent  Government  evaluators  and  gives 
the  Government  more  confidence  In  reported  test  results  than  might  other- 
wise be  expected.  To  some  degree  this  satisfies  the  previous  independent 
test  philosophy  objectives  with  minimum  test  hardware,  cost  and  time.  How- 
ever, the  contractor  Is  responsible  for  the  planning,  conduct  and  evalu- 
ation of  his  tests  regardless  of  where  the  tests  are  conducted.  As  a 
result,  the  contractor  test  plans  must  be  coordinated  with  concerned 
Government  agencies  to  Insure  that  all  of  the  required  test  objectives  are 
addressed  In  the  plan  and  that  the  tests  are  properly  designed  to  satisfy 
those  objectives.  Herein  lies  a problem.  If  the  contractor's  test  plan 
must  satisfy  the  objectives  of  all  concerned  Government  agencies  and  become 
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a major  part  of  the  materiel  developer's  Coordinated  Test  Plan,  the  coordi- 
nation must  be  accomplished  without  significant  Impact  on  the  contract 
cost  and  schedule.  This  can  become  quite  a challenge. 

The  relatively  new  Single  Integrated  Test  approach  employed  by  the 
Army  has  thus  made  the  contractor's  test  program  the  heart  of  the  Develop- 
ment Tests  (DT-II)  and  the  Coordinated  Test  Plan. 


Requirements  for  Contractor  Testing 

DOD  guidance  for  Major  System  Acquisition  process  indicates: 

"Contractors  shall  be  required  to  submit  firm  proposals  for 
full-scale  engineering  development  and  initial  production  upon 
completion  of  the  competitive  demonstrations  and  shall  be  pro- 
vided with  the  factors,  criteria  and  conditions  to  be  used  by 
the  DOD  in  the  evaluation  and  selection  of  a system  for  full- 
scale  engineering  development.  Specifications  and  standards  and 
a contract  data  list  shall  be  identified  and  tailored  by  the 
contractors  for  application  to  the  system  proposal  for  full- 
scale  development  on  the  basis  of  the  demonstration  and  validation 
results."5*9 

The  materiel  developer  Implements  the  DOD  guidance  by  issuing  a 
Request  for  Proposal  (RFP)  for  Engineering  Development  to  all  contractors 
who  successfully  participate  in  the  Demonstration  and  Validation  Program. 

The  RFP  includes  all  of  the  guidance  required  in  DOD  5000.2  plus  a general 
description  of  the  test  program  as  approved  in  DSARC  II.  Excerpts  from 
a typical  RFP  are  presented  In  Appendixes  A and  B to  illustrate  the  type 
of  guidance  provided  to  contractors  for  preparation  of  their  proposals.  It 
may  be  noted  that  the  RFP  excerpts  provide  considerable  guidance  on  testing 
philosophy  but  very  little  Information  on  Government  test  requirements  which 
must  be  Integrated  with  contractor  objectives  and  satisfied  by  contractor 
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testing.  Herein  lies  a potential  problem  area  which  will  be  addressed  later 
in  this  paper. 

Normally,  the  RFP  requires  that  the  winning  contractor  submit  his 
Engineer  Design  Test  Plan  {as  described  in  Appendix  B,  Data  Item  Number 
DI-T-190T7)  within  sixty  to  ninety  days  after  contract  initiation  for 
coordination  with  Government  test  agencies  and  approval.  When  approved, 
the  contractor's  test  program  automatically  becomes  part  of  the  materiel 
developer's  Coordinated  Test  Plan. 

The  coordination  and  approval  cycle  for  the  contractor's  test  program 
may  or  may  not  be  an  easy  task.  If  the  contractor  has  not  interpreted  the 
RFP  requirements  properly  such  that  his  test  plan  does  not  satisfy  the 
Government  test  objectives,  the  iterative  process  of  amending  the  con- 
tractor's test  program  begins.  As  previously  noted,  this  iterative  process 
must  be  accomplished  within  limited  bounds  and  without  significant  effects 
on  contract  cost  or  schedule. 

Thus  the  requirements  and  guidelines  for  contractor  testing  are 
established  and  levied  on  each  of  the  bidding  contractors  for  Engineering 
Development. 


SECTION  III 


OBSERVATIONS  AND  DISCUSSION 


Review  of  the  Present  Situati on 


The  present  trend  within  the  DOD  seems  to  emphasize  a "disengagement" 
policy  for  Project  Managers  which  allows  contractors  to  pursue  and  propose 
innovative  solutions  for  the  materiel  needs.  In  summary,  the  Project 
Manager  should  not  be  so  directive  that  he  inhibits  the  contractor's  ability 
to  develop  the  weapon  system.  At  what  point  then  should  the  Project  Manager 
start  to  direct?  He  cannot  relinquish  his  program  responsibility.  As  noted 
by  the  Logistics  Management  Institute: 

"In  common  with  the  way  a general  manager  must  operate,  the 
program  manager  relies  on  others  to  do  the  work.  But  he  cannot 
escape  the  responsibility  for  the  result.  If  he  is  responsible, 
he  must  be  satisfied  that  what  is  done  in  his  program  makes  sense 
to  him  and  is  consistent  with  his  plans.  If  he  cannot  be  persuaded 
that  it  is  right  for  his  program,  he  must  direct  it  to  be  done  the 
way  he  wants . 

Much  has  been  written  about  the  role  of  industry  and  the  re- 
lationship that  should  be  obtained  between  the  defense  program 
manager  and  his  industry  counterpart.  Much  has  been  said  about 
"disengagement"  - getting  out  of  industry's  hair  and  letting  them 
do  the  job  they  have  contracted  to  do.  The  goal  is  laudable  and, 
the  way  it  Is  stated,  the  idea  Is  entirely  consistent  with  good 
management  concepts.  But  the  ultimate  responsibility  for  a suc- 
cessful program  rests  squarely  on  the  Service  and  on  the  military 
program  manager  as  Its  agent.  The  program  manager  cannot  disengage 
In  any  literal  sense.  He  must  manage  contracted  work  In  Just  the 
same  sense  as  he  manages  all  other  parts  of  his  program.  More 
precisely,  in  this  case  he  manages  the  contractor  management  of 
his  program.  It  Is  not  a question  of  whether  he  manages;  It  Is 
only  a question  of  how  he  manages  - or  mismanages. "12, 5 


The  point  to  be  made  Is:  the  contractor(s)  must  be  given  the  freedom 
they  need  to  develop  the  weapon  system  and  the  associated  test  program(s); 
however,  the  Project  Manager  must  ascertain  if  the  proposed  test  program(s) 
sufficiently  satisfy  the  major  Government  test  objectives  - performance, 
safety,  reliability,  etc.  Alternatives  available  to  the  Project  Manager 
are  primary  issues  and  are  addressed  In  this  study  project. 

How  are  Coordinated  Test  Plans  for  weapon  systems  developed?  As  pre- 
viously noted  the  Project  Manager  develops  the  Coordinated  Test  Plan  for 
his  program  by  integrating  test  plans  and  requirements  from  the  contractor(s) 
and  Government  test  agencies. 

The  RFP  is  the  vehicle  by  which  the  program  manager  conveys  his  test 
requirements  to  contractors  who  in  turn  respond  with  contract  proposals. 
Through  the  "Source  Selection  Evaluation"  process  a contractor  is  then 
selected  for  Engineering  Development  and  his  proposed  test  program  becomes 
part  of  the  project  Coordinated  Test  Plan. 

So  what  is  the  problem?  The  problem  is  that  after  the  contract  is  signed 
and  the  Project  Manager  learns  more  and  more  details  about  the  contractor's 
test  program,  he  often  learns  that  the  contractor's  test  program  is  not  what 
he  had  envisioned  and  does  not  satisfy  the  Government's  need.  In  other 
instances,  the  source  selection  board  may  recognize  that  the  contractor's 
proposed  test  program  does  not  totally  satisfy  the  need;  however,  the 
proposal  may  still  be  considered  "responsive"  to  the  RFP.  Procurement 
regulations  and  procedures  do  not  readily  facilitate  resolution  of  this 
problem  without  "leading"  the  contractor  (which  is  not  allowed). 


14 


Some  of  the  major  concerns  expressed  by  persons  interviewed  during  this 

study  were: 

1.  The  required  test  effort  needs  to  be  clearly  defined  in  the  RFP. 

The  objective  of  the  RFP  is  to  provide  bidders  with  adequate  infor- 
mation and  guidance  in  a clear  and  logical  manner  to  elicit  proposals 
containing  all  the  information  needed  for  objective  evaluation.  The 
quality  of  the  RFP  guides  and  directs  the  type  of  proposal  responses 
to  be  received;  consequently,  contract  proposals  cannot  be  expected 
to  satisfy  all  of  the  Government's  test  objectives  if  the  RFP  is  not 
definitive. 

2.  Test  Programs  presented  in  contract  proposals  are  described  in  general 
terms  which  can  often  be  interpreted  differently  by  Government  and 
contractor  personnel.  Many  times  it  is  not  learned  until  after  the 
contract  is  signed  that  the  contractor's  test  program  does  not  include 
everything  the  program  manager  thought  it  included.  If  the  Project 
Manager  takes  action  to  correct  the  deficiencies  in  the  test  program, 
the  contractor  often  claims  that  the  increased  effort  is  "out-of-scope" 
for  the  contract  and  the  Project  Manager  suddenly  finds  himself  bar- 
gaining from  weakness.  Changes  to  the  test  program  can  significantly 
affect  cost  and  schedule.  The  Intent  here  is  not  to  imply  negligence 
on  either  the  part  of  the  contractor  or  the  Project  Manager,  but  to 
point  out  that  there  has  been  a communications  problem  in  the  past. 

This  problem  will  not  just  go  away  with  time.  It  must  be  recognized 
and  dealt  with  properly. 
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3.  We  need  better  management  discipline,  from  the  Project  Manager  stand- 
point, in  the  contract/procurement  process.  It  may  take  a little 
longer  to  sign  the  contract,  but  the  track  record  of  increased  cost 
and  management  headaches  resulting  from  poorly  defined  contracts  in- 
dicate that  time  spent  on  the  front  end  of  the  program  to  get  a good 
contract  is  time  well  spent  and  will  more  than  pay  for  itself  during 
performance  of  the  contract. 

Many  times  management  is  in  too  big  of  a rush  to  get  out  the  RFP 
and  to  sign  the  contract  once  the  proposals  are  received.  The  attitude 
is  that  details  of  the  test  program  can  be  worked  out  later.  But  as 
expressed  by  one  author  (a  Government  contractor)  - 

"The  old  expression  that  "haste  makes  waste"  was  never  more 
true  than  in  the  front  end  of  an  R&D  Program.  The  usual  reaction 
of  a program  manager  who  knows  that  money  is  time  and  people  is 
to  squeeze  them  both,  ultimately  making  the  mistake  of  assuming 
too  much.  He  assumes  that  a technique  that  worked  on  one  project 
will  work  on  another,  perhaps  taking  a component  or  subsystem  from 
a sister  project  without  bothering  to  test  it.  And  when  he  goes 
into  production,  the  Questions  he  glossed  over  during  R&D  come 
back  to  haunt  him".2*' 

The  message  here  is:  take  time  before  signing  the  contract  to  clarify 
the  contractor's  proposed  test  program,  to  understand  it,  to  agree  or 
disagree  with  it,  and  to  be  confident  that  the  Government  test  object- 
ives will  be  achieved. 

4.  In  the  competitive  arena  of  bidding  for  Government  R&D  contracts  where 
cost  is  a prime  determining  factor,  contractors  by  necessity  limit 
their  test  program  to  meet  the  minimum  requirements  of  the  RFP  (as 
they  understand  it).  Again  the  need  for  clearly  defined  RFP  require- 
ments Is  stressed.  There  is  a relatively  high  probability  that  the 
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contractor  and  the  Project  Manager  will  have  different  interpretations 
of  "minimum  RFP  requirements." 

5.  We  often  get  caught  up  in  "money  problems."  Testing  is  expensive  and 
contractors  can  only  do  so  much  with  the  available  money.  We  often 
find  ourselves  trading  off  testing  to  get  the  cost  down  to  "sell  the 
program".  Such  decisions  come  back  to  haunt  us  later  at  major  decision 
points  down  stream  when  we  have  to  explain  why  ^ufficent  testing  was  not 
accomplished  during  the  program. 

6.  There  seem  to  be  discontinuities  between  the  SSEB  evaluation,  negotia- 
tion agreements  and  the  contract.  First,  personnel  serving  on  the 
SSEB  will  be  more  highly  motivated  to  "do  a good  job"  if  they  know 

that  they  will  be  responsible  for  the  program  later  on.  Project  Manager 
personnel  should  be  utilized  to  the  maximum  practical  extent  on  the 
SSEB.  It  would  also  help  if  the  SSEB  Chairman  were  a member  of  the 
Project  Manager's  team.  We  need  more  continuity  between  the  SSEB 
and  the  project  management  team.  Second,  Issues  which  the  SSEB 
members  noted,  and  thought  were  resolved  during  negotiations,  somehow 
did  not  get  into  the  contract.  The  SSEB  members  need  to  participate 
in  contract  negotiations  to  assure  that  all  critical  issues  are  resolved 
and  affected  in  the  contract.  In  summary,  without  the  right  people  on 
the  SSEB,  contract  quality  often  gets  traded-off  and  the  result  is  a 
vague  contract  which  Is  difficult  to  manage, 

7.  Some  Government  personnel  over  emphasize  the  "disengagement"  policy 

as  noted  earlier  and  use  it  as  an  excuse  to  stay  out  of  the  contractor's 
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business,  let  him  propose  program  objectives  as  he  perceives  them, 
and  approve  the  proposal  as  submitted.  The  opinion  of  this  author, 
however,  is  that  such  a policy  results  in  less  management  and  more 
mis-management.  The  tax  payers  deserve  and  expect  more  for  their 
money.  The  Project  Manager  must  retain  the  responsibility  of  manage- 
ment, for  in  the  end,  he  is  the  one  held  accountable  for  program 
results. 

Have  Contractor  Master  Plans  Been  Adequate  in  the  Past 

As  one  might  expect,  the  quality  of  Contractor  Master  Test  Plans  in 
the  past  has  ranged  from  very  poor  to  very  good.  The  quality  of  the  plans 
seems  related  to  several  factors  such  as: 

1.  How  well  requirements  were  defined  in  the  RFP. 

2.  Communications  between  the  Government  and  contractor  personnel 
during  the  Valldation/Demonstratlon  phase  about  R&D  test  requirements. 

3.  Whether  the  bid  for  R&D  Is  sole- source  or  competitive. 

4.  Whether  or  not  the  contractor  has  previously  conducted  R&D  programs 
for  the  Army.  Testing  requirements  for  the  different  services  vary  somewhat 
because  of  the  varied  operational  environments  in  which  the  weapon  systems 
must  be  operated  and  supported.  Contractors  who  have  been  involved  in 
previous  Army  programs  generally  have  a better  appreciation  and  understand- 
ing of  Artty  peculiar  test  requirements. 

Considerable  literature  has  been  written  about  the  frustrations  of  the 
RFP  - proposal  cycle  and  hopefully  this  attention  will  result  In  better 
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Government/contractor  eomtiuni cations  in  the  future.  For  example,  BG  Esposito, 
USAF  Director  for  Procurement  Policy,  OASD  (I&L)  recognized  these  problems 
and  emphasized  concise  but  adequate  instructions  in  RFPs.  He  emphasized 
that  all  levels  of  management  should  assume  a major  role  to  bring  about  change 
in  the  acquisition  business.  LTG  Stewart,  U.  S.  Air  Force  Systems  Command, 
reinforced  the  need  for  change  when  he  wrote: 

"The  prototype  programs  emphasizing  streamlined  management, 
procurement  and  development  approaches,  provide  the  kind  of  environ- 
ment in  which  imaginative  ideas  can  surface,  be  applied  and  tested. 

A good  place  to  begin  improving  the  acquisition  process  is  clarifi- 
cation and  simplification  of  requests  for  proposals  (RFPs),  which 
would  provide  opportunity  to  overhaul  the  source  selection  process."** 

How  important  is  the  Contractor  Master  Plan  and  what  are  its  benefits? 
Planning  is  coming  to  grips  with  the  hard  details  of  program  execution.  It 
involves  the  examination  and  reexamination  of  the  problems  which  are  anti- 
cipated and  the  alternative  ways  in  which  these  problems  might  be  solved. 

Peter  Sandretto1  in  his  book  "The  Economic  Management  of  Engineering"  wrote: 

"After  all  has  been  said  and  done  about  systems  to  control 
engineering  costs  and  performance*  after  the  decision  is  made  to 
embark  on  a project*  it  is  the  project  plan  prepared  before  start- 
ing the  work  that  determines  to  a major  extent  the  outcome  of  a 
project  in  terms  of  time,  cost  and  technical  performance. "Z"9*1 

Sandretto  also  pointed  out  that  there  is  an  almost  unilateral  lack  of 
realization  that  the  project  plan  can  and  does  steer  the  project  away  from 
disaster  if  properly  planned  and  followed.  Evidence  of  this  lack  of  reali- 
zation (from  both  the  Government's  and  contractor's  point  of  view)  is  all 
too  evident  when  development  programs  get  into  trouble  as  a result  of 
inadequate  or  insufficient  testing.  As  previously  noted*  the  contractor's 
test  plan  becomes  the  heart  of  the  Project  Manager's  Coordinated  Test  Plan. 


The  importance  of  this  plan  cannot  be  overemphasized.  Some  contractor 
Master  Plans  have  been  adequate  in  the  past;  however,  the  "batting  average" 
needs  to  be  improved. 


SECTION  IV 


SUMMARY 


Summary  of  Conclusions 

The  following  conclusions  are  drawn  as  a result  of  this  study: 

1.  The  D00  and  DA  directives  and  regulations  have  clearly  established 
the  requirement  and  issued  guidelines  for  preparation  of  the  program 
manager's  Coordinated  Test  Plan. 

2.  Program  managers  are  following  the  official  guidance  and  preparing 
Coordinated  Test  Plans  to  include  contractor  and  Government  tests 
with  integrated  test  objectives  as  applicable  to  minimize  test 
hardware,  cost  and  schedule. 

3.  Under  the  Single  Integrated  Test  Policy,  contractor  testing  has 
become  the  heart  of  the  Coordinated  Test  Plan.  This  fact  has  and 
still  is  causing  some  difficulties.  RFPs  and  contracts  are  some- 
times vague  and  general  in  nature  and  the  program  manager  doesn't 
realize  until  he  is  well  into  the  contract  that  the  contractor's 
test  program  is  Inadequate  or  does  not  satisfy  all  of  the  Govern- 
ment test  objectives. 

4.  The  RFP  - proposal -contract  process  leaves  a lot  to  be  desired. 

The  problems  are  complicated  by  many  factors  and  there  does  not 
appear  to  be  a simple  solution;  however,  constructive  recommen- 
dations are  included  In  this  report. 
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5.  The  importance  of  the  Master  Test  Plan  needs  to  be  stressed  at  all 
levels  of  management  (Government  and  contractor)  to  assure  that  it 
receives  proper  priority  and  that  the  plan  is  followed  once  it  is 
established.  A poor  plan  may  very  subtly  allow  the  development  to 
head  for  disaster  from  which  recovery  will  be  very  costly  in  both 
time  and  money.  On  the  other  hand,  an  orderly,  well  constructed 
test  plan  will  assist  in  identifying  problem  areas  early  in  the 
program  when  recovery  is  not  so  costly.  It  will  provide  necessary 
data  to  the  program  manager  so  that  he  can  make  rational  trade-off 
decisions  between  cost,  schedule  and  performance  as  the  development 
program  progresses. 

6.  Development  and  testing  make  up  an  iterative  process  where  the 
contractor  designs  his  hardware,  evaluates  the  performance,  makes 
changes  as  necessary  and  re-tests  for  performance.  DOD  policy  trends 
seem  to  imply  Project  Manager  "disengagement”  which  allows  the  con- 
tractor to  pursue  innovative  solutions  for  weapon  system  needs. 
However,  the  program  manager  cannot  Ignore  the  fact  that  he  is 
ultimately  responsible  for  system  development  and  that  at  some 
strategic  point  in  time  he  must  bring  the  innovative  process  under 
his  control  and  lead  the  development  into  production.  The  Coordi- 
nated Test  Plan  should  clearly  Identify  that  strategic  point  in  the 
development  program  and  clearly  define  follow-on  testing  which 

must  be  successfully  accomplished  to  demonstrate  acceptable  cost, 
performance,  safety,  reliability,  maintainability,  etc. 
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Recommendations 


As  a result  of  this  study  and  the  conclusions  presented  herein,  the 
following  recommendations  are  submitted  for  consideration: 

1.  That  program  managers  take  the  necessary  time  to  develop  an  RFP 
which  clearly  states  the  minimum  test  requirements  and  identifies 
Government  test  objectives.  A recommended  approach  would  be  to 
Include  potential  R&D  contractors  in  the  CTP  coordination  process 
during  the  Validation  and  Demonstration  phase,  to  develop  a relatively 
detailed  Coordinated  Test  Plan,  and  to  Include  the  CTP  in  the  RFP. 
Utilizing  this  approach,  the  contractor(s)  should  clearly  understand 
the  Government's  test  objectives  and  minimum  test  requirements.  As 

a minimum,  it  is  recommended  that  the  CTP  contain  formal  contractor 
component  qualification,  system  performance/reliability  tests,  environ 
mental  qualification  tests  as  applicable  and  Independent  Government 
tests  including  sample  sizes,  test  conditions  and  general  test  pro- 
cedures. This  allows  contractor(s)  the  flexibility  to  conduct 
necessary  hardware  design  selection  and  development  testing  prior 
to  formal  qualification:  it  provides  him  the  information  necessa»y 
to  schedule  and  budget  hardware  and  support  costs  required  for  the 
“total"  R&D  test  program:  It  Is  also  consistent  with  the  current 
trend  In  DOD  policy. 

2.  That  Project  Manager  personnel  occupy  key  positions  on  the  Source 
Selection  Evaluation  Board  (SSEB).  Persons  are  more  highly  moti- 
vated when  they  realize  that  they  will  be  managing  the  program  In 
the  future  and  any  mistakes  or  deficiencies  they  accept  on  the  SSEB 
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will  be  their  headaches  In  the  future.  (There  needs  to  be  continuity 
between  the  contract  selection  and  contract  performance.) 

3.  Project  personnel  who  serve  on  the  SSEB  should  participate  In  con- 
tract negotiations  to  assure  that  their  Inputs  get  Into  the  contract. 
Negotiating  personnel  who  do  not  understand  the  relevance  or  Import- 
ance of  technical  detail  are  more  likely  to  overlook  or  eliminate 
Important  aspects  of  the  SSEB  evaluation. 

4.  Do  not  hurriedly  put  an  RFP  out  for  consideration,  rush  the  evalu- 
ation, or  sign  a contract  which  has  not  been  throughly  considered. 
Along  the  same  line,  do  not  sign  a contract  that  has  deficiencies 
with  the  Idea  that  problems  will  be  worked  out  later.  It  Is  not  a 
simple  matter  to  correct  deficiencies  once  a contractor  and  all  of 
his  sub-contractors  have  been  "turned  on"  and  It  will  Invariably 
Increase  cost.  Time  taken  at  the  beginning  of  the  program  to  get 
contract  matters  In  order  will  save  even  more  time  during  contract 
execution  and  result  In  an  earlier  completion  date  with  fewer 
management  headaches. 

5.  Baseline  Development  Specifications  for  all  configuration  Items 
should  be  developed  during  the  Valldatlon/Demonstratlon  phase  and 
Included  In  the  RFP,  As  with  the  test  plan,  generating  Baseline 
Development  Specifications  after  contract  Initiation  may  result  In 
considerable  problems,  require  that  some  tests  be  repeated,  and 
delay  progress  of  the  Program. 

In  summary,  define  clearly  what  you  need  before  you  sign  the  contract 
and  accept  nothing  less.  Plan  and  manage  the  total  test  program,  don't  let 
It  manage  you. 
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APPENDIX  A 


EXCERPTS  FROM  A TYPICAL 

REQUEST  FOR  PROPOSAL  FOR  ENGINEERING  DEVELOPMENT 


C-27  MASTER  PROGRAM  SCHEDULE 

Attachment  30  is  the  Governments  Master  Program  Schedule  (MPS)  which  depicts 
certain  key  milestones.  The  contractor  shall  provide  with  his  proposal  an 
expanded  MPS  which  clearly  portrays  scheduled  time  for  accomplishment  of  all 
tasks  required  by  this  RFP  and  including  the  reviews  required  by  J-4  and  the 
contractor  proposed  performance  milestones  C-36.  The  contractors  proposed 
Master  Program  Schedule  shall  support  delivery  of  hardware  as  specified  in 
paragraph  H-2,f. , and  the  functional  and  allocated  base  line  specifications 
as  specified  in  Section  F,  0001AF2.4. 1.4.2  and  Data  Item  DI-E-1104A,  sequence 
numbers  009  and  00B. 

0001AE  (U)  Test  & Evaluation 

0001AE1  Development  Test  (DT): 

DT  will  consist  of  the  following  Phases:  Engineering  Design  Test  - Contractor 
(EDT-C),  Engineering  Design  Test  - Government  (EDT-G),  Prototype  Qualifi- 
cation Test  - Contractor  (PQT-C),  Prototype  Qualification  Test  - Government 
(PQT-G),  and  Production  Validation  Test  - Contractor  (PVT-C)  as  defined  In 
AR  70-10. 

The  contractor  shall  submit  a Contractor  Master  Test  Plan  to  the  Government 
in  accordance  with  DI-T-1900.  This  test  plan  shall  be  an  update  of  the  pre- 
liminary Contractor  Master  Test  Plan  submitted  as  a part  of  the  proposal  and 
shall  present  a general  description  of  each  major  test  indicating  hardware 
quantities  and  utilization,  test  objectives,  data  requirements,  test  sites, 
list  of  GFP,  special  test  equipment,  and  contractor  support  as  applicable 
for  each  major  test.  The  Contractor  Master  Test  Plan  shall  present  sufficient 
detail  to  Indicate  that  the  contractor  has  planned  a test  program  which  will 
enhance  development  on  a timely  basis  and  demonstrate  hardware  performance 
and  reliability  at  all  assembly  levels.  Log  books  shall  be  provided  with  all 
test  Items  for  configuration  Identification  In  accordance  with  DI-E-1101B. 
Maximum  utilization  of  Government  test  facilities  Is  required  unless  proven 
to  be  uneconomical.  The  contractor  shall  furnish  and  maintain  special  test 
equipment  to  support  the  test  program. 


25 


0001AE1 . 1 Engineering  Design  Test  (EDT); 

The  contractor  shall  plan  and  conduct  an  orderly  test  program  starting  with 
component  selection  and  evaluation  and  progressing  through  total  system 
demonstration.  This  test  program  shall  demonstrate  that  the  system's  per- 
formance meets  the  requirements  of  MIS  23103.  The  contractor's  detailed 
test  plan  shall  include  as  a minimum  the  following  areas:  structural,  electri- 
cal and  performance  characteristics  of  components,  subassemblies,  assemblies, 
and  total  system;  human  factors  engineering;  safety;  reliability  and  maintain- 
ability; mutual  interference;  countermeasures;  and  nuclear/electromagnetic 
radiation.  The  contractor  shall  conduct  tests,  maintain  all  development  hard- 
ware, analyze  data  and  correct  deficiencies. 

0001AE1.1.1  Component/Subsystem  Performance  Test  (EDT-C): 

The  contractor  shall  plan  and  conduct  performance  tests  on  selected  piece 
parts  including  component,  subassemblies  and  assemblies  of  each  end  item. 

Each  generation  of  hardware  shall  be  evaluated  under  expected  operating  con- 
ditions to  include  nominal  as  well  as  selected  natural  and  induced  environ- 
ments as  identified  In  the  Environmental  Criteria  specified  in  MIS  23103. 

After  designs  for  hardware  have  been  selected,  critical  components  shall  be 
delivered  to  the  Government  for  evaluation.  These  components  are  identified 
as  Contractor  Furnished  Equipment  (CFE)  elsewhere  in  the  SOW.  In  addition, 
the  contractor  shall  place  In  bonded  storage  an  adequate  quantity  of  engineer- 
ing critical  components  {of  the  configuration  of  the  10th  serially  numbered 
guided  flight  missile)  necessary  for  the  contractor  to  test  and  evaluate  per- 
formance degradation  under  the  items  critical  environment (s)  for  the  purpose 
of  establishing  SHELF  LIFE  base  line  data. 

0001AE1 .1.3  Missile  Flight  Tests  (EDT-C): 

Missile  flight  tests  shall  be  conducted  starting  as  early  as  practical  and 
continue  through  Engineering  Development  (ED)  to  demonstrate  hardware  per- 
formance as  predicted  from  component/subsystem  performance  tests  and  at  all 
operating  conditions  delineated  In  MIS  23103.  Missile  flights  will  be  ini- 
tiated as  early  as  practicable  during  ED  and  continue  In  a logical,  sequential 
manner  to  demonstrate  the  performance  characteristics  under  varying  operational 
modes.  Flight  tests  shall  be  conducted  as  the  configuration  progresses  toward 
final  design  and  shall  demonstrate  performance  and  reliability.  The  effects 
of  selected  natural  and  induced  environments  shall  be  investigated  during  the 
flight  test  program  to  obtain  confidence  that  the  hardware  will  pass  the  sub- 
sequent System  Qualification  Tests  (para  0001AE1.4).  All  technical  require- 
ments shall  be  demonstrated.  The  contractor  shall  Include  data  requirements 
and  accuracy  requirements  In  the  test  plan.  All  missile  flights  shall  be 
conducted  at  a Government  furnished  test  facility. 
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0001AE1.3  Component  Qualification  Test  (PQT-C): 

The  contractor  shall  plan  and  conduct  an  orderly  test  program  on  components 
selected  during  EDT  to  demonstrate  performance  and  reliability  of  the  design 
in  simulated  and  actual  operational  environment.  The  contractor  shall  update 
the  Component  Qualification  Test  Section  of  the  Contractor  Master  Test  Plan 
to  identify  the  components  which  will  undergo  qualification  test  and  shall 
submit  the  revised  plan  to  the  Government  in  accordance  with  DI-T-1900  at 
least  30  days  prior  to  initiation  of  the  Component  Qualification  Test  Program. 
This  test  program  shall  be  to  determine  if  the  components  meet  their  per- 
formance requirements  while  operating  under  or  after  being  subjected  to  the 
environmental  extremes  necessary  for  system  functions  as  identified  in  MIS 
23103.  A sequential  type  test  program  shall  be  employed  utilizing  minimum 
sample  sizes  and  a typical  environmental  sequence  as  will  be  encountered  in 
operational  use.  Test  results  will  also  be  evaluated  to  determine  reliability 
data  Including  failure  rates,  parameter  variations  with  environment,  mode  of 
failure  and  life  expectancy  with  regard  to  anticipated  operating  conditions. 

0001AE1.4  System  Qualification  Test  (PQT-C): 

The  contractor  shall  plan  and  conduct  an  orderly  test  program  on  assemblies 
selected  during  EDT  to  demonstrate  performance  and  reliability  of  the  design 
in  simulated  and  actual  operational  environment.  The  contractor  shall  update 
the  System  Qualification  Test  Section  of  the  Contractor's  Master  Test  Plan 
to  Identify  the  assemblies  (missiles,  launchers,  ground  support  equipment, 
etc.)  which  will  undergo  qualification  test  and  shall  submit  the  revised 
plan  to  the  Government  in  accordance  with  DI-T-1900  at  least  30  days  prior 
to  initiation  of  the  System  Qualification  Test  Program.  Tests  shall  be  con- 
ducted to  determine  if  end  items  meet  their  performance  requirements  while 
operating  under  or  following  exposure,  as  appropriate,  to  a reasonable  com- 
bination of  environments  specified  for  the  system.  End  items  utilized  In 
this  test  shall  consist  of  hardware  configurations  which  demonstrated  accept- 
able performance  in  the  Component  Qualification  Test,  para  0001AE1.3.  A 
sequential  environmental  test  approach  shall  be  utilized.  The  performance 
requirements  and  environmental  levels  are  defined  in  the  environmental  criteria 
of  MIS  23103.  This  test  shall  consist  of  a minimum  of  24  missiles  (8  each 
for  the  High,  Ambient,  and  Low  Temperature  sequence).  Data  generated  during 
this  program  will  also  be  utilized  to  determine  system  reliability.  Defi- 
ciencies discovered  during  this  test  phase  shall  be  corrected  by  the  con- 
tractor prior  to  hardware  being  released  for  subsequent  Government  test. 

Test  shall  be  conducted  at  a Government  furnished  test  facility. 


APPENDIX  B 


Data  Item  Descriptions: 

DI-T-1900  Coordinated  Test  Plan 

DI-T-1901  Engineer  Design  Test  Plan 

DI-T-1903  Part,  Component  or  Subsystem  Plan(s) 

from  the  Department  of  Defense  AMSDL,  Acquisition  Management  Systems  Data 

3 

Requirements  Control  List. 
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DATA  ITEM  DESCRIPTION 


iBeHrme*TiQ»  now). 


AGENCY 


NUMBER 


I.  TITLE 

Coordinated  Test  Plan 


Army 


DI-T-1900 


i.  dcscription/purposc 

The  Coordinated  Test  Plan  presents  a general  outline  for  a total  test 
for  a weapon  or  equipment  system,  for  the  testing  and  evaluation 
conducted  during  development  to  assure  that  an  Item  meets  Army 
requirements,  AR  70-10,  para  I-! . This  will  be  used  In  preparation 
of  detail  test  plans,  schedules  and  procedures. 


“ZSpnoVTCoTtE" 
15  Dec  69 


OTTiee  OK  PRIMARY 
RES«ONMBIL(tT 


USAMC 


I.  DDC  AKQUIRSO 


I.  APPWOVAU  UMITATtg* 


7.  APPLICATION/INYEARKI-  ATIONSNiP 

Applicable  to  any  system  or  equipment  requiring  test.  Normally  de- 
livered in  the  conceptual  or  early  definition  phase  for  approval  by  the 
Government. 

DI-S-I819,  Contractor  Recommended  Support  Plan 
DI-R-1730,  Reliability  Program  Plan 
DI-R-1740,  Maintainability  Program  Plan 
DI-H-1313,  Human  Factors  Engineering  Test  Plan 
DI-T-1904,  Integrated  Test  Plan 
DI-A-1012,  Documents  Required  by  National  Ranges 
See  DI-H-1312  or  DI-H-1313  (when  specified);  Dl-H-1320;  DI-R-1700 
or  DI-R-1701  for  respective  program  requirements  of  Human  Factors, 
Safety  and  Product  Assurance.  See  DI-R-1700  or  DI-R-1701  for 
acceptance  inspection  test  program  for  deliverable  hardware. 


*.  ^BPKRKNCB^rMMutotMY**  clfdin 

AR  70-10 
AR  70-27 


Men.  NuwtiRU) 


10,  PREPARATION  INSTRUCTION# 


The  Coordinated  Test  Plan  shall  contain  plans  and  procedures  to  assure  achievement  of  operational  re-1 
qulrements  delineated  by  QMR's,  SDR's  and  Technical  Requirements  of  the  contract.  It  shall  describe 
the  experimental  design  for  obtaining  data,  delineate  functions  and  tolerances,  and  establish  and  • 
explain  standards,  tests,  associated  analysis  and  other  means  for  constituting  adequate  proof  that 
acceptable  performance,  including  all  military  effectiveness  factors,  can  be  achieved  In  operational 
use.  Test  plans  shull  cover  ail  related  equipment  to  be  tested  for  determining  adequacy  and  compati- 
bility of  the  weapon  (or  equipment)  system  and  shall  Include  but  not  be  limited  to  the  following: 

1 . Tests  to  prove  adequacy  of  individual  parts  of  Individual  items 

2.  Tests  to  prove  adequacy  of  subassemblies  and  assemblies  of  various  subsystems  of  the  system 

3.  Functional  tests  of  various  subsystems,  Individually  and  in  combination,  to  determine  effects  of 
individual  subsystems  on  each  other  In  the  system 

4.  Tests  of  packaged  equipment  to  determine  adequacy  for  transport,  storage,  handling,  environ- 
mental conditions  and  other  specified  requirements 

5.  Tests  to  determine  compliance  with  reliability,  maintainability,  safety  and  human  factors  engi- 
neering requirements  of  Dl-H-)3)2y  1313;  Dl-H-1320  and  DI-R-1700,  1701 

6*  Functional  tests  of  complete  weapon  or  equipment  system  to  determine  satisfactory  operation  and 
performance  In  accordance  with  requirements 


DD-S2M664 


29 


rabb 


OR 


2.  »Aa— 


I 


1 

i 

t 

DI-T-1900 

preparation  Instruction*  (Continued) 

location  and  duration  of  tests,  extent  of  Government  and  contractor  participation  and  test  instrumen- 
tation required  shall  be  indicated.  Requirements  for  National  and  Service  Ranges  facilities  and 
services  shall  be  documented  in  National  Range  Documentation  DI-A-1012.  Use  of  any  other  Govern- 
ment equipment  or  systems  required  to  support  the  tests  shall  be  described.  Procedures  for  processing 
arid  handling  the  data  obtained  shall  reflect  maximum  use  of  Government  facilities. 


DATA  ITEM  DESCRIPTION 


IDENTIFICATION  NOtSI. 


ACENCV  ( NUMBER 


Engineer  Design  Test  Plan  , Arm/ 


DESCRIPTION/  PUPPOSE 

The  Engineer  Design  Test  (EDT)  Plan  describes  the  contractor's 

proposal  for  the  conduct  of  the  EDT  portion  of  the  Coordinated  Test  |v  StaiSlSJ.JirTY*'"' 


, Arm  / DI-T-1901 


4.  APPROVAL  OAT C 

15  Dec  1969 


Plan  Di-T-1900  (AR  70-10)  for  a weapon  or  equipment  system.  The 
purpose  of  the  EDT  is  to  collect  design  dota,  confirm  preliminary 
concepts,  and  determine  compotability  of  components. 


USAMC 


S.  OOC  REQUIRED 


I.  APPROVAL  LIMITATION 


?.  APPLICATION/IN  TEMPEL.  ATIONSHIP 


Data  obtained  from  the  EDT  is  used  for  initial  design  of  a prototype. 

The  EDT  may  include  the  Human  Factors  Engineering  tests  described  t rrrinciin  im.  j 
in  DI-H-1313,  Human  Factors  Engineering  Test  Plan.  This  data  item 
is  included  as  a part  of  Dl-T -1900, Coordinated  Test  Plan,when  that  ^ tq.jq 
data  item  is  a program  requirement.  AMCR  70-7 


UCU.  MUMBEMM 


10.  PREPARATION  INSTRUCTIONS 


The  EDT  Plan  for  a weapon  (or  equipment}  system  shall  describe  the  methods  and  techniques 
for  obtaining  data  and  shall  establish  and  explain  the  standards,  tests,  associated  analysis  and 
other  means  that  will  constitute  adequate  proof,  upon  completion  of  the  development  phase, 
that  an  acceptable  level  of  performance  can  be  achieved  by  the  system.  The  test  plan  shall 
apply  to  all  equipment  that  makes  up  the  weapon  (or  equipment)  system.  It  will  Identify  tests  to 
be  conducted  for  determining  compliance  with  stated  technical  and  operational  military 
requirements  and  the  individual  equipment  specifications.  Maximum  use  of  Government  owned 
facilities  and  instruments  will  be  stipulated. 

When  required,  EDT  Plans  shall  be  prepared  for  use  In  tho  development  of  ports,  components  or 
subsystems  to  a point  of  qualification  and  use  in  higher  assemblies  within  a weapon  (or  equipment) 
system.  The  plans  shall  include,  but  not  be  limited  to,  philosophy,  method  of  test,  schedules 
of  performance,  procedures  for  check  out,  acceptance  limits  for  performance  capabilities, 
requirements  and  characteristics,  description  of  test  equipment  to  be  ustd,ond  list  of  required 
C? > v*rnrnenT owned  equipment  and  facilities.  The  plans  must  demonstrate  confidence  that  areas 
of  coverage  ore  adequate  and  that  dato  generated  wilt  assure  that  objectives  of  performance  cast 
be  validated. 


DD'R*1664 


DATA  ITEM  DESCRIPTION 


1.  TITcE 

Pori,  Component  or  Subsystem  Test  Pfon(s) 

3.  OESCRlPTlON/PURPOie  “ “"™  ~ " 

The  Part,  Component  or  Subsystem  Test  Plan(s)  presents  the  philoso- 
phy, test  logic,  sequence  and  test  schedules  and  test  procedures  to 
be  utilized  in  the  development  of  parts,  components  or  subsystems 
to  a point  of  qualification  and  use  in  higher  assemblies  within  a 
weapon  (or  equipment)  system.  If  may  be  used  in  conjunction  with 
production  and  use  phase  tests. 


*.  APPLICATlON7lNTCRR*L  AYION9MIP 

This  plan  provides  detailed  test  guidance  on  each  test  contained  in 
DHM900,  Coordinated  Test  Plan. 


PREPARATION  *N*TNUCT|ON» 


\2. IDENTIFICATION  NOfSl. 


AGENCY 

NUMBER 

Army 

DI-T-1903 

4.  APPROVAL  OAT* 


15  Dec  1969 

9.  OFFICE  Or  PRIMARY 
RC9PON94HCftV 

USAMC 


A.  DOC  ACQUIRED 


(.  APPRO V AW  WINMTATIOM 


».  REP E A EHCE3 <55355? 5»  cSiSTip* 


3-10 
AMCR  70-7 


MC«W  NUMRKA4M 


The  Port,  Component  or  Subsystem  Test  Plants)  shall  Include,  but  not  be  limited  to,  philosophy, 
method  of  tost,  schedules  of  performance,  procedures  for  checkout,  acceptance  limits  far  perform- 
ance capabilities,  requirements  and  characteristics,  description  of  test  equipment  to  be  used,  and 
list  of  required  government  equipment  and  facilities. 


The  plan  must  demonstrate  confidence  that  areas  of  coverage  ore  adequate  and  that  data  generated 
will  assure  thot  the  objectives  of  performance  can  be  validated. 


DD 1664 
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APPENDIX  C 


ISP  INTERVIEW  SHEET 


Name:  

Project:  

Position:  

Other:  

1.  Explain  the  nature  and  objectives  of  this  Study. 

2.  Explain  that  this  interview  is  on  a non-attribution  basis  and  that 
information  provided  will  be  used  in  context  while  protecting  its  source. 

3.  Question; 

A,  What  Is  the  status  of  your  Coordinated  Test  Plan? 

O.  How  detailed  were  your  instructions  in  the  RFP? 

C.  Did  the  contract  proposals  provide  sufficient  detail  for  evaluation 
and  understanding  of  the  proposed  test  program? 

D.  How  soon  after  contract  initiation  did  the  contractor  update  his 
Master  Test  Plan? 

E.  Did  you  have  any  significant  problems  getting  a coordinated  position 
within  the  Army  concerning  adequacy of  the  contractor  test  plan? 

P.  Were  all  wend  item"  included  in  the  contractor  test  plan  {containers* 
maintenance  and  test  equipment*  training  equipment,  etc.}? 
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G.  Were  you  satisfied  with  the  contractor's  test  plan  concerning: 

(1)  Component  testing? 

(2)  Sample  Sizes? 

(3)  Environmental  testing? 

(4)  Qualification  testing? 

(5)  Reliability  testing? 

H.  If  you  were  not  satisfied  with  any  of  the  above,  what  action  was 
taken  and  what  was  the  program  impact? 

I.  Do  you  think  Government  instructions  regarding  testing  were  adequate 
in  the  RFP? 

J.  Did  you  think  the  contractors  understood  the  RFP  requirements  and 
responded  properly? 

K.  Do  you  feel  that  the  Government's  approach  to  the  RFP/SSEB/Negotiation 
Process  enhances  or  complicates  the  procurement  process? 

L.  Do  you  have  any  reconmendatlons  for  future  procurement  actions? 
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